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SUMMARY 


The capability for simulating a teleoperator/robotic system is being developed 
to support research and technology programs related to remote space operations. A 
preliminary version of this simulation has been used to examine the effects of con- 
trol modes, a visual proximity cue, and time delays on the manual control of a probe- 
in-hole alignment task using a simulated f ive-degree-of-f reedom manipulator and 
computer-generated visual displays. 

Three visual displays were simulated. Two displays represented television pic- 
tures taken from movable cameras, the locations of which could be controlled by the 
subjects. The third display represented a television (TV) picture from a fixed cam- 
era looking along the axis of the probe. The subjects preferred to position the two 
movable cameras to provide front and side orthographic views. The subjects had lit- 
tle apparent difficulty in assimilating the visual information, despite having to 
select the view to be presented on the single TV monitor. 

Two control modes were simulated: joint-by- joint proportional-rate control and 

resolved-rate control with discrete translational rates and proportional rotational 
rates. Proportional rates were commanded by using a two-axis rotational hand con- 
troller and a joint-axis switch matrix. The time required to complete the simulated 
task was more than three times longer using joint-by- joint control than when using 
resolved-rate control, and alignment errors were significantly larger when using 
joint-by-joint control. 

The proximity display was a cross icon superimposed on the probe TV picture, 
with the length of each arm of the cross proportional to the distance from the simu- 
lated task board. Rotational errors were indicated by unequal arm lengths. Thus, 
the proximity-sensor display enabled subjects to separate rotational errors from 
displacement (translation) errors. Therefore, when the proximity display was used 
with re solved- rate control, the simulation task was trivial. 

Time delays up to 2 sec were simulated. For time delays of 0.25 sec and longer, 
subjects replaced continuous control with a move-and-wait control strategy. Dis- 
playing a cue that the input had been received was helpful for longer time delays. 

The time required to perform the simulation task increased linearly with time delay, 
but time delays had no effect on alignment accuracy. 

Based on the results of this simulation, several future studies are recommended. 


INTRODUCTION 

The Langley Research Center is supporting the development of teleoperator and 
robotics technology that will be required for remote space operations such as satel- 
lite servicing, inspection, recovery, fuel transfer, and construction. This research 
and technology program makes use of (1) a Teleoperator/Robotic Systems Simulation 
(TRSS) which will model in software actual or conceptual teleoperator/robotic sys- 
tems; (2) an Intelligent Systems Research Laboratory (ISRL) having a network of dis- 
tributed microconputers and miniconputers interfaced to vision systems, speech syn- 
thesis and recognition systems, and dual six-degree-of-f reedom manipulators equipped 



with end effectors having touch, force, torque, and proximity sensors; and ( 3 ) a 
reconfigurable control station with multipurpose displays and controllers with which 
man can interface with the simulation and/or the laboratory. Presently, each of 
these facilities is under development and in various stages of completion. Capabili- 
ties will evolve and improve during the course of the research programs. 

This report describes the initial teleoperator simulation which examined the 
effects of control modes, a proximity- display method, and time delays for a probe-in- 
hole alignment task using a simulated f ive-degree-of-freedom manipulator. 


SYMBOLS 


Values are given in SI Units and, where considered useful, also in U.S. Custom- 
ary Units. Measurements and calculations were made in U.S. Customary Units. 

J 5x5 Jacobian matrix 

J 3 3x3 submatrix of J 

K empirical constant used to limit arm extension 

b^/L 2 length of manipulator upper arm and lower arm, respectively, cm 

T 3 x 3 matrix transforming velocity commands from end-effector coordi- 

nates to X,Y,Z system 

Tj completion time using joint-by-joint control 

completion time using resolved-rate control 

time delay, sec 

components of orthogonal coordinate system located at manipulator 
shoulder, cm 

components of end-effector velocity commanded from cockpit, cm/sec 
= ® 3 ' 

angle-of-pitch rotation at manipulator shoulder, elbow, and wrist, 
respectively, deg 


At 

X,Y,Z 


X ,Y ,Z 
C C C 


® 1 , ' ® 2'®3 


= 0 +0 +0 


0 

C 

a 

2 

X 

4-1, 4>3 


1 2 3 

pitch rate of end effector commanded from cockpit, deg/sec 
standard deviation, cm or deg 
chi-square distribution 

angle-of-yaw rotation at manipulator waist and wrist, respectively, deg 
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(b = (L + i 

*(\)^ yaw rate of end effector commanded from cockpit, deg/sec 

A dot over a symbol indicates differentiation with respect to time- 


DESCRIPTION OF SIMULATION 

The teleoperator/robotics simulation is programmed on a CDC^ CYBER 175 computer 
operating in real time at 32 iterations per second, which is interfaced to an ADAGE 
GPS/340 graphics system and a single-place general-purpose cockpit. Manipulator 
equations, relative geometry, and data recording are handled by the CYBER 175- The 
ADAGE draws high-speed, monochrome vector (stroke) displays by using data from the 
CYBER. The vector display is converted to a 525-line television (raster) format by a 
vidicon camera facing the ADAGE-display cathode-ray tube (CRT) . Figure 1 shows an 
ADAGE drawing of the simulated manipulator - 


Simulated Manipulator and Task Board 

The simulation models a f ive-degree-of-f reedom manipulator having yaw at the 
waist, pitch at the shoulder, pitch at the elbow, and pitch and yaw at the wrist. 
Figure 2 illustrates the manipulator configuration. 

Each axis is assumed to respond as a second-order linear servo system; the damp- 
ing ratio is 0.7 in all axes; and the natural frequency is 5 rad/sec in the first two 
joints (waist and shoulder) and 10 rad/sec in the other three joints (elbow and 
wrist). The upper and lower arms are each 43 cm (17 in.) long, and the probe 
attached to the wrist is 15 cm (6 in.) long. 

The order of rotation and the arm lengths are representative of the UNIMATE 
PUMA^ manipulators installed in the ISRL. However, the image of the simulated manip- 
ulator is more representative of the Protoflight Manipulator Arm (PFMA) at the Mar- 
shall Space Flight Center. (See ref. 1.) 

The simulation also models a Gl-cm^ (2-ft^) planar task board containing five 
"holes.” (See fig. 3.) The subject's task was to align the probe accurately with 
the center of each hole. Forces and moments at contact and conpliance were not 
modeled, but probe angular misalignment and the displacement of the tip of the probe 
with respect to the center of the hole were coitputed. 


Simulator Cockpit 

Figure 4 is a photograph of the simulator cockpit. This general-purpose cockpit 
is equipped with aircraft instruments, a three-axis proportional hand controller, a 


^CDC: Registered trademark of Control Data Corporation. 

^UNIMATE PUMA; Registered trademark of Unimation, Inc. 
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throttle lever, mode- select switches, and indicator lights. This simulation used the 
following equipment: 

(1) A 33-cm (13-in.) black-and-white television (TV) monitor mounted at the top 

center of the cockpit 

(2) A three-axis hand controller for yaw and pitch commands 

(3) A speed-brake switch located on the throttle lever for fore and aft transla- 

tion commands 

(4) A mode- select switch panel on the left side of the instrument panel. Mode- 

select options are described in the next section. 

(5) Two three-position center-loaded toggle switches, located below the mode- 

select panel, for up-down and right-left translation commands 

(6) Two lights, below the toggle switches, to indicate when the computer was 

limiting manipulator motion as the arm neared full extension 

^ light directly below the TV monitor to indicate satisfactory probe align- 
ment at contact 

The other switches, indicators, and instruments in the simulator cockpit were not 
used. 


DISPLAY AND CONTROL MODES 

The display options and the control modes were selected by using the mode-select 
panel located on the left side of the instrument panel. This 4x4 matrix of back- 
lit switches is illustrated in figure 5. Only 10 switches were used in the 

simulation. 


Camera and Display Control 

Three TV cameras were simulated; camera 1, camera 2, and the probe TV camera. 
Three switches in the first column of the mode- select panel were used to select the 
view to be displayed on the monitor. Cameras 1 and 2 could be moved by the opera- 
tor. The probe TV camera was fixed, looking along the axis of the probe. A symbol 
representing the line of sight (axis of the probe) was superimposed on the probe-TV 
visual scene. Control of cameras 1 and 2 was enabled by selecting the camera with 
one of the switches in the first (left) column of the mode-select panel, and then 
selecting the "TV"- control-mode switch in the second column. In this mode, the three 
translation (on-off) switches commanded translation of the TV camera (up-down, right- 
left, and fore-aft) with respect to TV-camera axes. The simulated TV camera would 
translate at 1 m/sec (39 in/sec) as long as a translation command was applied. The 
TV-camera pan (yaw) and tilt (pitch) were commanded by the rotational controller at a 
maximum rate of 15 deg/sec. Controller roll input had no effect. Initially, one 
camera was at a location above the manipulator and task board, and the other camera 
was looking in from the side. 

The "Reset TV" switch was included as a camera-control option because, ini- 
tially, there was concern that subjects might have trouble assimilating the visual 
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information or that they might move a TV camera to an undesirable or confusing loca- 
tion; thus, some means was needed for the subjects to restore a familiar visual envi- 
ronment. Pressing the "Reset TV" switch would cause the selected TV camera to return 
to its initial location. However, when testing began it was apparent that the visual 
scene did not confuse or disorient the subjects. Thus, the "Reset TV" option, 
although available, was not needed or used. 


Proximity-Sensor Display 

The availability of a simulated proximity display was one of the parameters in 
the study. Previous studies at the Jet Propulsion Laboratory (JPL) (refs. 2 and 3) 
have shown that proximity sensors can be incorporated in an end effector and can 
supply useful range and alignment information. For the simulation, it was assumed 
that four proximity sensors were located around the probe to provide range informa- 
tion within 15 cm (6 in. ) of the task board. The simulated sensor outputs were 
represented by four bars located around the probe line-of-sight symbol. When the 
probe was perpendicular to the task board, the display would be a cross, with size 
proportional to range. Figure 6 shows the proximity display indicating the probe 
rotated (yawed) to the right. The proximity display was available only with the 
"Probe TV" mode selected and the "Proximity display" switch on. (See fig. 5.) 


Input-Confirmation Display 

A small arrow was displayed in the lower right corner of the TV monitor during 
control inputs. This display (fig. 7) confirmed that the computer had received the 
input. A double arrow indicated a translation input, and a single arrow in the 
direction of motion indicated a rotational input. The arrow also provided an indica- 
tion of the duration of the control input- It was not subject to the transport 
delays in the visual scene. The subjects felt that the indicator was most useful 
with the longer time delays for confirming that the computer had received the command 
and that a result would eventually appear. Although the input-confirmation display 
was probably not necessary when there was no transport delay, the display was not 
distracting and, therefore, it was used throughout the study. 


Individual Joint Control 

Selecting the "Arm" control mode instead of the "TV" mode on the mode- select 
panel (fig. 5) enabled the operator to control the manipulator arm joints. However, 
since there were five manipulator joints and only two available controller commands 
(yaw and pitch), the subject had to select the desired joints by using the "Shoul- 
der," "Elbow," and "Wrist" mode switches. Selecting "Shoulder" enabled the subject 
to command yaw at the manipulator waist and pitch at the shoulder. Selecting "Elbow" 
enabled yaw at the waist and pitch at the elbow. Selecting "Wrist" enabled yaw and 
pitch commands to the wrist. The rate of rotation was proportional to the hand- 
controller deflection. Maximum angular rates were 30 deg/sec at the waist, shoulder, 
and elbow and 20 deg/sec at the wrist. 


Resolved-Rate Control 

Rotational rates at manipulator joints can be mathematically transformed and 
expressed as a combination of linear and angular velocities referenced to a 
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particular coordinate system. The coordinate system is often located at the wrist or 
in the end effector or tool. In resolved-rate control, linear and angular rates are 
commanded in an orthogonal axis system, usually in the wrist or end effector (in the 
probe for this simulation), and the inverse transformation is used to compute corre- 
sponding joint rates. Resolved-rate control is useful because it, in effect, decou- 
ples the manipulator motion by enabling manual commands to cause orthogonal end- 
effector motions. Resolved-rate control in several axis systems is the primary man- 
ual control mode for the Shuttle Remote Manipulator System, with joint-by- joint con- 
trol as backup mode. (See ref. 4. ) 

The simulated resolved-rate control mode was referenced to an axis system fixed 
in the probe. The subject could command yaw and pitch rates at the wrist and could 
command linear velocities of ±5 cm/sec (±2 in/sec) in the probe axis system. 
Resolved-rate control was available when "Arm" and "Wrist" modes were selected. (See 
fig. 5. ) However, in addition to the rotational controller commanding wrist angular 
rates, the translation command switches used for TV-camera control could be used to 
command linear velocities of the probe. (Up-down and right-left motions were 
commanded by switches on the instrument panel, and fore-aft motion was commanded by 
the speed-brake switch. ) For example, if a forward command were made with the speed- 
brake switch, the computer would automatically compute the joint rotational rates 
required to cause the probe (and wrist) to move forward along the axis of the probe 
at 5 cm/sec (2 in/sec). 

The resolved-rate equations used in this simulation were derived from differen- 
tial position changes conputed at each 1/32-sec time interval. Similar equations, 
but of simpler form, are obtained from position derivatives rather than differen- 
tials. The derivative form of the resolved-rate equations has been used in subse- 
quent simulations and is presented in the appendix. 


TIME DELAYS 

Two approaches for real-time remote control of a vehicle in the Earth's orbit 
are possible. One approach is direct line-of-sight control, either from the ground 
or from another spacecraft such as the Shuttle or a space station. A single ground 
station could be in direct contact with a spacecraft for only a fraction of its 
orbit. If control were from another spacecraft, a similar situation would occur 
unless the orbital periods of the two were matched. Time delays would not be signif- 
icant with line-of-sight control, but limiting control to certain time windows would 
reduce the usefulness of a remote system. 

A system of orbiting relay satellites, such as the Tracking and Data Relay Sat- 
ellite System (TDRSS), is an alternative approach. A TDRSS satellite could relay 
data between the remote vehicle and the manned control station. T^en TDRSS is com- 
pleted, three satellites will provide global coverage. (See ref. 5.) However, use 
of a relay satellite for real-time control of a remote spacecraft presents several 
potential problems: First, it has not been done before. Second, the communications 

bandwidth would limit the rate and/or fidelity of television pictures, probably 
requiring onboard video preprocessing. Third, data formatting and transmission could 
introduce time delays. Since round-trip delays using a satellite relay are uncer- 
tain, a time delay of up to 2 sec was an independent simulation variable. 

Time delays At ofO, 0.25, 0.50, 1.0, and 2.0 sec were simulated. Half of the 
time delay was inserted by passing the control and mode select signals through a 
holding array prior to the equations of motion in the GDC CYBER 175 computer. The 
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other half of the time delay was inserted by passing signals from the CYBER into a 
holding array before going to the ADAGE graphics system. In the cockpit, when the 
operator made a control input, the resulting motion would be seen At seconds 
later. Similarly, a mode change (e.g. , from camera 1 to camera 2) would also be 
delayed At seconds. 

As noted earlier, the input-confirmation display was never delayed. This was 
equivalent to superimposing on the (delayed) television picture a graphics display 
driven directly by control inputs. 


PROCEDURE AND ANALYSIS 

The simulation was conducted as two separate studies. Five subjects, A to E 
aged 17 to 25, participated. The first study was a 4 x 2 x 2 factorial design with 
three replications. Four subjects (A to D) flew each control mode ( joint- by- joint 
and resolved-rate) and two proximity display modes (off or on). The time delay was 
zero. The order of presentation was joint-by- joint/display off, joint-by- joint/ 
display on, re solved- rate/di splay off, and re solve d-rate/di splay on. This order was 
selected because it involved increasing simulation sophistication and enabled the 
entire simulation to be learned in a logical sequence. 

The second study wasa4 x2 x5 factorial design with three replications. Four 
subjects (B to E) flew each control mode with five time delays (0, 0.25, 0.50, 1.0, 
and 2.0 sec). Three subjects had participated in the first study. The proximity- 
sensor display was operational for all runs. Two subjects began with the joint-by- 
joint control mode; the other two subjects began with the resolved-rate control mode. 
Time delays were presented in a random order. 

Subjects were instructed that the objective was to align and insert the probe 
into each hole on the task board (fig. 3), in a given sequence, starting and ending 
at the same hole. A different path was followed for each replication, as shown in 
figure 8, and the paths were presented in random order. Subjects were instructed 
that although time to complete the task would be recorded, accurate alignment was 
more important than time. If the probe reached the hole with a tip-displacement 
error greater than the radius of the hole (1.3 cm (0.5 in.)), the subject would get a 
visual indication of impact. (See fig. 9.) If an impact occurred, the subject was 
to withdraw the probe, realign, and make a successful insertion before proceeding to 
the next hole. Alignment errors at insertion and at impact were recorded. 

In addition to recording the time required to complete the task, the number of 
impacts, and the alignment accuracy, the total number and duration of manipulator 
control inputs were recorded for each control 'axis. Inputs were not recorded during 
control of the simulated TV cameras, but final camera locations were recorded. 

An analysis of variance (ref. 6) was performed on the data. Total task time was 
based on three replications. Angular misalignment and displacement errors for suc- 
cessful insertions were analyzed based on 18 insertions (6 holes x 3 replications). 
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RESULTS AND DISCUSSION 


Effects of Control Mode and Proximity Display 

Table I shows the mean alignment error (angular and radial displacement) of the 
probe tip for 18 insertions (6 holes x 3 replications) for each subject in the first 
simulation (no time delay) . Thble II shows the mean, root mean square (rms) , and 
chi-square (x ) value of the error over insertions and subjects. The rms errors are 
presented instead of the standard deviation (a) because much of the data show strong 
positive skewness. A chi-square goodness-of-fit test (see ref. 6 ) rejected the 
hypothesis that the data came from a normally distributed population at the 0.01 
level (table II) for all except radial-displacement errors associated with joint-by- 
joint control. Figures 10 and 11 show histograms of the alignment errors. The data 
in tables I and II indicate the following reTsults: 

(1) The proximity display had a large effect on angular-alignment accuracy, but 

effect on radial—displaceinent— alignment accuiracy* 

(2) The control mode had a large effect on both angular and radial-displacement- 

alignment accuracy. 

(3) Differences in subjects had little effect. 

When the proximity display was used with resolved-rate control, subjects consis- 
tently achieved precise alignment. The display and the resolved-rate control mode 
were well-matched because the display decoupled or separated the rotational- and 
translational-alignment errors and the decoupled controls made corrections easy. The 
subject would rotate the probe until the arms of the cross icon were of equal length, 
then translate the probe until the center line of the probe (fig. 6 ) was centered on 
the hole, and then translate forward for insertion. 

Without the proximity display, subjects used the same technique but the rota- 
tional alignment was made by using the two orthogonal TV views. The TV views gave 
ess-precise cues so that the rotational-alignment errors were larger. Insertion was 
still performed by placing the probe center line on the hole and translating 

require the proximity display, and the data in table I confirm 
that the proximity display had no effect on displacement errors. 

In the joint-by -joint control mode, the proximity display was much less helpful. 
Despite the precise cues provided for angular-alignment error, the subjects could not 
effectively decouple the motions because rotation of any joint resulted in both a 
rotation and a translation of the probe with respect to the task board. Because of 
this, subjects chose two methods of aligning the probe for insertion by using joint- 
by-joint control. One method was to minimize alignment errors a short distance in 
front of the hole, and then to accept the resulting errors on insertion. The other 

method was to deliberately establish small errors so that they would go to zero on 
insertion. 

These methods were not conpletely successful, as indicated in table III by the 
number of impacts. The proximity display had no effect on the number of impacts in 
the joint-by- joint control mode. With the resolved-rate control mode, there were 
five impacts without the proximity display but only one impact when using the 
di splay . 
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The time required to complete the task (table IV) shows control-mode effects 
clearly because the time required to move the probe between holes as well as the 
alignment time is included. Generally, the time required to complete the task in 
joint-by-joint control mode was two to three times longer. Table IV also shows that 
subject B spent two to three times longer than subject A. However, since the sub- 
jects were instructed that time was not critical, no operational conclusions should 
be drawn. 

Table V shows the average number and duration of control inputs during a repli- 
cation. Some subjects often used joint-by-joint control to move the arm close to the 
task board before beginning the sequence of alignment tasks using resolved-rate con- 
trol, probably because large rapid movements could be made faster by using joint-by- 
joint control. The proximity display apparently had no effect on the inputs, but the 
number of inputs appears directly related to the time required to complete the task. 
However, the simulated task enabled subjects to use very few inputs in the resolved- 
rate control mode. Since the task board was assumed planar, after once having 
aligned the probe with the first hole, the subject had only to translate the probe to 
subsequent holes. A three-dimensional task board would have required translation and 
rotation at each hole. 


Effect of Control Mode and Time Delay 

In the second study, which was a 4 x 2 x 5 factorial design, four subjects per- 
formed the alignment task with both control modes, with proximity display on, and 
with five time delays (-0, 0.25, 0.50, 1.0, and 2.0 sec). Three of the four subjects 
(B, C, and D) had participated in the first study (no time delays), and their results 
were used for the zero-time-delay case. 

Table VI shows the mean angular- and radial-alignment errors for each subject, 
and the mean and rms errors over all subjects. The time delays had no effect on the 
alignment errors. It should be noted that without the proximity display, the angular 
errors would probably have been greater, as was shown in the first study. 

Table VII presents the time required to complete the simulated task. The table 
shows a large variation with subjects using the joint-by-joint control mode. Some of 
the variation is due to the control technique used by each subject and the time spent 
in final alignment. Another source of variation is the time required to repeat an 
alignment when an impact occurred. Table VIII presents the number of impacts during 
the test. The number of impacts appears to depend on the subject (and control tech- 
nique) and control mode, but not on the amount of time delay. 

The data in table VII indicate that the time required to complete the task was 
strongly influenced by both the time delay and the control mode. The time to com- 
plete the task, averaged over the subjects, is plotted in figure 12. Considering the 
variation in the coirpletion times for individual subjects (table VII) , the average 
completion time is remarkably linear for both control modes. The linear relation 
between task-conpletion time and time delay is consistent with an earlier study by 
Ferrell (ref. 7) for a master-slave manipulator task. Ferrell found that the rela- 
tion between coitpletion time and delay was essentially linear for a given task, the 
main effect being due to the time spent waiting for feedback. Ferrell postulated 
that if the operator controlled the remote device by turning motors on and off rather 
than by using the master-slave control, a similar move-and-wait strategy would be 
used, and a similar linear relation would occur between conpletion time and time 
delay. 
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This simulation relates directly to Ferrell’s prediction. The joint-by- joint 
control mode was analogous to an on-off control where the rate was selected by the 
subject. The resolved-rate control mode was a rate-controlled task. 

The solid lines in figure 12 are least-squares fits to the experimental data. 
The completion time using joint-by- joint control Tj is given by 

Tj = 460 + 399 At 

the con^letion time using resolved-rate control T^^ is given by 
Tj^ = 137 + 145 At 

and the dashed line in figure 12 is given by 



The good fit of the dashed line to the experimental data in figure 12 indicates 
that not only were the average completion times directly related to time delay but, 
with any delay, the conpletion times for the two control modes also appear to be 
directly related. The relation between time delay, control mode, and task-completion 
time found in this study suggests that possible relations might be observed in other 
studies. 


Input Display 

The confirmation display (fig. 7) was used in both studies and, therefore, its 
value cannot be quantified. Subjects commented that the display was not distracting 
and that it was most useful with the longer time delays for confirming that the com- 
puter had received the command and that a result would eventually appear. 

An unexpected benefit of the display was shown when a wire behind one of the 
input switches broke during the simulation. The sudden loss of confirmation of that 
input suggested an open signal line and resulted in rapid fault isolation. 


Location of Simulated Cameras 

An advantage of a coirputer -generated display, such as that used in this simula- 
tion, is the ability to position the lookpoint (simulated-camera location) at any 
desired location. Subjects were encouraged to move the simulated cameras extensively 
during the familiarization phase. As a result, two camera locations were subjec- 
tively identified which were, somewhat surprisingly, satisfactory to all the sub- 
jects. Although subjects could reposition the cameras at any time, they were moved 
in only 7 percent of the data runs. One camera was positioned 87° laterally and 
4® vertically with respect to the center of the task board. Camera pan and tilt 
angles were slightly different (96® and -3®, respectively) in order to include the 
manipulator arm as well as the task board in the 60® camera field of view. The view 
from this location (fig. 13(a)) was almost down the side of the task board. It pro- 
vided horizontal- and vertical-displacement cues, but no lateral-displacement cues. 
The other camera location was above the task board looking down, 9® laterally and 
70® vertically with respect to the center of the task board, and sufficiently far 
back that the shoulder of the manipulator was visible. (See fig. 13(b).) The 
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corresponding camera pan and tilt angles were 1° and 72°, respectively. This view 
provided good lateral-displacement cues, but poor vertical and horizontal cues. 

Although subjects could move the simulated cameras at any time, in only 11 of 
the 156 tests was a camera moved from the initial location. In nine tests, the 
"side" camera was positioned within the range from 79° to 88° laterally and 0° to 
6° vertically. In seven tests the "overhead" camera was positioned within the range 
from 8° to 18° laterally and 68° to 71° vertically. 

Thus, throughout the simulation the subjects used essentially orthogonal views 
from the two cameras to position the manipulator. This is consistent with results of 
a recent study by Winey (ref. 8) which involved a computer graphic simulation of a 
seven-degree-of-f reedom slave manipulator controlled by an actual master controller 
having force-reflecting capability. Shadows, multiple views, and proximity indica- 
tors were evaluated to determine their effectiveness in giving depth information. In 
a task of grasping fixed and moving simulated spheres, Winey (ref. 8) found that 
front and side orthographic projections showed the best performance. Three of the 
four subjects preferred the orthographic display because they felt it gave the clear- 
est detail. However, reference 8 notes that the best depth indicator would probably 
be a combination of the front and side views with a shadow. 

In this simulation, none of the subjects reported any difficulty in assimilating 
the data from the three simulated TV scenes (including the probe view) presented on a 
single monitor, or in manually selecting the desired scene. 


POSSIBLE FUTURE STUDIES 

As was noted earlier, this simulation used the initial version of a 
Teleoperator/Robotic System Simulation (TRSS), which will evolve in sophistication to 
support the technology required for remote space operations. In the course of this 
simulation, a number of follow-on studies were suggested, some of which are discussed 
as follows: 

(1) Simulation of a three-dimensional task: The simulated two-dimensional task 

board made the alignment task trivial for a subject using resolved-rate control. A 
three-dimensional task would be more realistic. 

(2) Reduced bandwidth display: For a space task it is likely that the return 

signal will be bandwidth limited. The bandwidth can be reduced by reducing the tele- 
vision frame rate, reducing the television resolution, limiting the shades of gray, 
using onboard signal processing (e.g. , multiplexing), and/or using onboard image 
processing (returning only significant features of the scene, such as edges, corners, 
or motions). Studies (refs. 9 and 10) have examined some of these methods for remote 
control of underwater manipulators, and some of these results may be applicable to 
space operations also. 

(3) Simultaneous video display: An alternative to selecting a single video dis- 

play manually from several options would be to present all views simultaneously. 

This would reduce the manual work load, but it might increase the mental work load as 
the subject tried to assimilate visual inputs, particularly if the multiple views 
were presented at lower frame rates. 

(4) Six degrees of freedom with end effector: For the simulated alignment task, 

only five degrees of manipulator freedom were simulated. Generally, a manipulator 


11 



will have six or seven degrees of freedom plus an end effector. A sixth degree of 
freedom, corresponding to rotation about the axis of the probe, and a parallel-jaw 
end effector are being added to the simulation. This will enable simulation of 
grasping and turning as well as alignment tasks. 

(5) Laboratory coitparison: The joints, links, and end effectors of the 

simulated manipulator represent the actual manipulators in the Langley Intelligent 
Systems Research Laboratory (ISRL). A study comparing a simulated manipulator task 
with the same task performed by using actual hardware would validate the simulation 
and would be a base line for future studies. 

(6) Type of visual presentation: In this study, none of the subjects reported 

any difficulty in assimilating the data from the three simulated TV scenes. In the 
study reported in reference 8, the front and side orthographic projections were dis- 
played side by side on a single monitor. Other studies have compared single-camera, 

earner a, and stereoscopic display systems and have found that the preferred sys- 
tem depended upon task, camera location, and display bandwidth. By coupling the 
manned control station in ISRL, which has a stereo-vision system, to the ISRL manipu- 
lators through the simulation, it will be possible to compare direct vision, mono and 
stereo TV, and mono and stereo conputer-graphics displays in a manipulator control 
task. 


(7) Sensor models and force/torque control: The manipulator in ISRL has several 

sensor feedbacks that are not presently modeled in TRSS, including motor currents (or 
borques) at each joint, three forces and three torques at the manipulator wrist, and 
two forces and two torques at the jaws of the end effector. These feedbacks make it 
feasible to command forces and torques and to control the cortpliance of the end 
effector or tool. These sensor signals and control modes should be modeled in TRSS 
in order to simulate general tasks such as touch, grasping, and insertion. The abil- 
ity to simulate a generic task such as insert-push-turn performed under manual con- 
t-fol, or under automatic control with manned supervision, is a goal in the 
development. 


CONCLUDING REMARKS 

A simulation has been conducted to examine the effects of two control modes 
(joint-by- joint and resolved-rate), a visual-proximity-cue display, and time delays 
(up to 2 sec) on the control of a f ive-degree-of-freedom manipulator performing a 
probe-in-hole alignment task. A high-speed conputer-graphics system generated the 
simulated television (TV) scene for three cameras, two of which could be moved in 
translation and rotation by the subject. However, the subject had to select the 
scene to be presented on a single TV monitor. 

Results showed that all subjects preferred to position the two movable cameras 
to supply essentially orthogonal views, with one camera looking in from the side of 
the task and the other looking down from a location above and behind the manipulator 
A nominal camera location was identified during familiarization runs and, although 
subjects could reposition the cameras at any time, they were moved in only 7 percent 
of the data runs. Subjects had no difficulty selecting and using the single display 
with multiple simulated cameras. 

As e>q)ected, subjects preferred the resolved-rate control mode over the joint- 
by-joint control mode. Not only were alignment errors lower with resolved-rate 
control, but also less time and fewer inputs were required to acconplish the task. 
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thus indicating a lower subject work load. If a metric for mental and physical work 
load could be developed, it would be a useful tool for future studies. 

The proximity display provided accurate range and angular-alignment information 
from four assumed, equally spaced, simulated sensors. The proximity display was 
superimposed on the view from a simulated TV looking along the axis of the probe (the 
third TV viewing option) , and it enabled subjects to differentiate probe -displacement 
errors from angular errors. Use of the proximity display had no effect on the 
radial-alignment (displacement) errors, thus suggesting that the displacement-error 
cues provided by the two orthogonal TV views were as effective as the displacement 
cues from the proximity display. However, the angular-alignment errors were signifi- 
cantly lower (with either control mode) when the proximity display was used. 

1*He proximity display and the resolved— rate control mode together made the 
alignment task almost trivial, because the display resolved rotational and transla- 
tional errors and the control mode enabled them to be controlled separately. The 
task was made easier by the assumption of a planar task board. Additional studies 
with a nonplanar task would be desirable. 

Time delays were simulated as a transport delay, with half the delay between the 
control input and the simulated manipulator response and half the delay between the 
manipulator response and the visual display. The delays forced the subjects to adopt 
a move— and— wait control strategy for all simulated delays (0-25 sec and longer), but 
delays had no effect on alignment accuracy. 


The major effect of a simulated time delay was to increase the time required to 
complete the task. The average time required to complete the task increased directly 
with time delay, with the task time for the joint-by- joint control mode increasing at 
a greater rate. The relation between the task time for the two control modes was 
almost constant - about 3.3 times greater for the joint-by- joint control mode. The 
direct relation between time delay, control mode, and task— completion time should be 
tested in subsequent studies. 

A useful feature of the simulation was an unobtrusive indicator which was dis- 
played during manual-control inputs. The indicator was superimposed on the visual 
display and confirmed the type (translation or rotation) and direction of the 
input. It was most useful with longer time delays because it verified that the com- 
puter had received the input, thereby assuring eventual response. 


Langley Research Center 

National Aeronautics and Space Administration 
Hampton, VA 23665 
July 25, 1983 
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APPENDIX 


RESOLVED-RATE EQUATIONS 

In resolved-rate control/ the operator specifies the rate of end-effector motion 
(translational and rotational) and a control algorithm transforms these commands into 
rate commands for the individual joints- The resultant end-effector motion is the 
sum of all the contributions made by the joints- By expressing end-effector velocity 
as the sum of joint contributions, a system of equations is obtained that relates 
end-effector rates to joint rates- The system of equations is linear and can be 
solved by matrix methods. 

The simulated manipulator had five degrees of freedom; therefore, five end- 
effector motions (three translations and two rotations) could be commanded. Thus, 
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end 

^3 


(A1 ) 


where 6 


is the angular rate at the manipulator waist; 


angular (pitch) rates at the shoulder, elbow, and wrist, 
the yaw rate at the wrist. (See fig. 1.) 


0 , 0 , and 0 

1 2 3 

respectively; and 


are 

(L is 
^3 


Inverting the Jacobian matrix [J] would give an exact solution for the resolved 
rates, but performing a 5 x 5 matrix inversion 32 times per second for the real-time 
simulation was undesirable. A nimiber of methods have been proposed for speeding up 
the solution of the matrix-inversion problem. One approach is to separate the trans- 
lational and rotational motion, resulting in two smaller matrices which can be easily 
inverted. This approach is discussed in reference 11 and was used in the simulation. 
It was assumed that only the first three joints affected the linear motion; that is, 
resolved-rate translational commands were referenced to the wrist. In addition. 
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6^/ and 0^ were sequential rotations so that the shoulder, elbow, and wrist, 

respectively, were coplanar. Thus, 
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Then, 


0 ^ = 0 - (0 + 0 ) 
3 C 12 


6 = i i 

'+'3 


(A3) 


(A4) 


(A5) 


(A6) 


• • 

where 0 and 4) are end-effector angular-rate commands from the rotational hand 
C C 

controller in the cockpit. 


The terms of the matrix were referenced to an X,Y,Z axis system, illus- 

trated in figure 14, centered at the manipulator shoulder, with the Z-axis aligned 
with the 4)^-axis of rotation and rotating about Z so that the X-Z plane contained 
the upper and lower arms and wrist. The position of the wrist was 


X = cos 0^ + cos(0^ + 0^) 


(A7) 
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Y = 0 


(A8) 


Z = -L^ sin 9^ - sin(0^ + G^) 


The velocity of the wrist, then, is 

X = -L sin 0 9 - L sin( 0 + 0 ) ( 0 + 0 ) 

1 112 12 12 

Y = X4;^ 

Z - -L cos 0 0 - L cos( 0 + 0 )(9 + 0 ) 

1 112 12 12 


and 


X cos{ 9 + 0 ) - Z sin( 9 + 9 ) 

12 12 

0 = 

1 L sin 0 

1 2 


(e, * 9^, 


-X cos 0 + Z sin 0 

1 1 


L sin 9 
2 2 


(A9) 


(A10) 


(All) 


(A12) 


(A13) 


(A14) 
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X L COS 6 + L cos( 9 + 0 ) 

1 12 12 


(A16) 
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The resolved-rate translation commands 
transformed to the X,Y,Z axis system by 


from the cockpit were 
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where K is an enpirical multiplier, 

sin a 
0 

cos a 

and 

a = 01 + 02 + 63 


COS cos a 

|t| = I sin 

-cos (l;^ sin a 


-sin 4)3 cos a 
cos 4)3 
sin a sin 


(A18) 


(A19) 


When 02 = 0, equations (A13), (A14), and the J 3 matrix (eq. (A2) ) are singu- 
lar. A matrix inversion would still be possible by using a pseudoinverse matrix 
solution, but in the simulation the singularity was avoided by limiting 0 by using 
the empirical multiplier K in equation (A17) where ^ 


K = 1 


when I 0 I > 25° or 0 0 >0 and 

' 2 ' 2 2 


K = 



20 ° 


(A20) 


when I 0 I <25° and 0 0 < 0. 

'2' 22 


The effect of this limit was to reduce slightly (by 0*2 percent) the maximum 
extension of the arm and to slow the motion of the arm as it neared maximum exten- 
sion. This was not a problem for the subjects since all of the simulated task board 
was within the reach envelope. 
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TABLE I.- ALIGNMENT ERRORS WITHOUT TIME DELAYS 


Angular error by subject 


Angular error, deg, for proximity 
display - 


Subject 


Joint-by- j oint 
control mode 


Resolved-rate 
control mode 


A 

3.9 

2.5 

B 

5.5 

1.5 

C 

4. 1 

4.0 

D 

5.0 

.9 

Mean • • • • 

4.6 

2.2 

rms 

6.0 

4.3 


(b) Radial-displacement error by subject 


Radial-displacement error, cm, for 
proximity display - 


Subject 


Joint-by- j oint 
control mode 


A 

O 

• 

0^ 

k 

B 

.79 

C 

.50 

D 

.62 

Mean • • • • 

0.63 

rms 

0.72 


Resolved-rate 
control mode 


9 























TABLE II 


ALIGNMENT ERRORS WITHOUT TIME DELAYS 


(a) Angular error for all subjects 


Statistical 

values 

Angular 

error, deg, for proximity 
display - 

Off 

On 

Off 

On 

Joint-by- joint 
control mode 

Resolved-rate 
control mode 

Mean 

4.6 

2.2 

msm 

0.4 

rms 

6.0 

4.3 

WM 

0.5 

2 

Y value 

37.7 

123.2 

25.1 

50.5 

Xo.oi 

16.8 

15. 1 

15. 1 

13.3 

significant at 0.01 •••• 

Yes 

Yes 

Yes 

Yes 

a 

1 

3.9 

3.8 

0.7 

0.3 


(b) Radial-displacement error for all subjects 


Statistical 
va lues 

Radial-displacement error, cm, for 
proximity display - 

Off 

On 

Off 

On 

Joint-by- j oint 
control mode 

Resolved-rate 
control mode 

Mean 

0.63 

0.62 

0.30 

0.28 

rms 

0.72 

0.72 

0.37 

0.40 

9 

Y value 

11.4 

15.7 

23.4 

76.6 

xioi 

18.5 

16.8 

16.8 

15.1 

Significant at 0.01 .... 

No 

No 

Yes 

Yes 

a 

0.35 

0.36 

0.21 

0.29 


2 






















TABLE III.- NUMBER OF IMPACTS 


Subject 


Total 


Number 

of inpacts with proximity 
display - 

Off 

On 

Off 

On 

Joint-by- joint 

Resolved-rate 

control mode 

control 

mode 

2 

0 

1 

0 

2 

4 

0 

0 

3 

2 

3 

0 

2 

3 

1 

1 

9 

9 

5 

1 


TABLE IV. - AVERAGE TASK TIME FOR THREE REPLICATIONS 


Subject 

Average task time, sec, for 
proximity display - 

Off 

On 

Off 

On 


Joint-by -joint 
control mode 

Resolved-rate 
control mode 

A 

300 

226 

99 

116 

B 

602 

629 

296 

201 

C 

319 

388 

213 

125 

D 

423 

437 

129 

113 

Mean 

411 

420 

184 

139 















TABLE V.- AVERAGE NUMBER AND DURATION OF CONTROL INPUTS FOR A SINGLE REPLICATION 


[One replication is equivalent to six alignments] 


Control 

applied 

Average niomber and duration of control inputs with proximity display - 

Off 

On 

Off 

On 

Number 

Duration, 

sec 

Number 

Duration, 

sec 

Number 

Duration, 

sec 



Number 

Emration, 

sec 

Joi 

nt-by- joint control 

mode 

Re solved- rate 

control 

mode 

Waist (yaw) 

60.0 

33.0 

48.5 

31.0 

1.4 

0.2 

0.8 

0.2 

Shoulder (pitch) 

64.7 

34.3 

52.2 

30.5 

0.5 

0.2 

0.7 

0.3 

Elbow (pitch) 

110.3 

70.8 

101.3 

63.9 

2.2 

0.9 

0.9 

0.7 

Wrist (yaw) 

27.2 

29.3 

30.6 

27.2 

6.5 

3.6 

9.3 

2.9 

Wrist (pitch) 

25.8 

19.5 

40.8 

28.0 

12.8 

5.2 

16.4 

5.3 

Fore and aft velocity 





20. 1 

14.6 

18.3 

15.6 

Lateral velocity 





33.3 

26.1 

28.8 

25.5 

Vertical velocity • 





27.5 

26.6 

33.4 

26.2 

Ttotal 

288.0 

186.9 

273.4 

180.6 

104.3 

77.4 

108.6 

76.7 




TABLE VI. -ALIGNMENT ERRORS WITH TIME DELAYS 


Sxibject 


Mean • • • . 
riTTS .... 


(a) Angular error by subject 

Angular error, deg, for time delay of 
0 sec 0.25 sec 0.50 sec 1.0 sec 


Resolved-rate control mode 



CO 

. 

0.48 

.15 

.20 

.55 

.21 

.04 

.04 

.31 

0.23 

.40 

0.32 


2.0 sec 



Joint -by -joint 

control mode 


B 

1.57 

1.29 

1.40 

2. 13 

1.69 

C 1 

3.96 

2.27 

2.30 

1.91 

.56 

D 

.88 

1.34 

1.33 

1.69 

1.88 

E 

1.62 

1.87 

1.98 

1.35 

1.69 

Mean • • « « 

2.01 

1.69 

1.75 

1.77 

1.46 

rms 

4.18 

2.10 

2.09 

2.19 

1.97 



Mean 
ritis . 


(b) Radial -displacement error by subject 



Radial 

-displacement error, cm. 

for time delay of - 

0 sec 

0.25 sec 

0.50 sec 

1.0 sec 

2.0 sec 


Joint-by-joint control mode 



0.56 

0.64 

0. 78 

0.99 

0 

.84 

.53 

.46 

.54 

.71 

.80 

.74 

.80 

.85 

.75 

.65 

.36 

.43 

.46 

.41 

.46 

0.55 

0.59 

0.66 

0.72 


.69 

0.65 

0.68 

0.76 

0.81 

1 


.77 


Resolved-rate control mode 


B 

0.20 

0.17 

0.18 

C 

.23 

.19 

.27 

D 

.27 

.25 

.24 

E 

.14 

.19 

.20 


rms 










































TABLE VII.- AVERAGE TASK TIME WITH TIME DELAYS 



Joint -by -joint control mode 


B 

629 

630 

633 

895 

1147 

C 

388 

484 i 

552 

977 

1842 

D 

437 

809 

943 

930 

1139 

E 

419 

356 

400 

672 

912 

Mean . • • • 

468 

570 

632 

869 

1260 


Resolved-rate 

control mode 


B 

201 

259 

194 

263 

380 

C 

125 

211 

201 

300 

582 

D 

113 

179 

184 

221 

329 

E 

130 

137 

190 

242 

468 

Mean • • • • 

142 

196 

192 

257 

440 
























L-82-839 

Figure 1.- Cortputer -generated image of simulated manipulator. 
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L-82-841 

Figure 3.- Simulated task board. 


28 




Camera 1 


Arm 


Camera 2 

TV 

Probe TV 

Proximity 

display 



Reset TV 


Shoulder 


Elbow 


Wrist 
















L-82-1 1,111 

Figure 6.- Simulated proximity display showing lateral (yaw) angular misalignment. 
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L-82-11,108 


(a) Lateral-translation indicator. 


Figure 7.- Simulated display showing control confirmation in lower right corner. 
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I/-82-11, 1 10 


(b) Pitch-up indicator. 
Figure 7.- Concluded. 
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Figure 9«- Impact display. 


Ii-82-4109 
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(a) Camera at side of task board* 


L-83-4247 


Figure 13*- View from initial simulated camera locations. 


41 




L-82-1 1 ,107 


(b) Camera above task board. 


Figure 13.- Concluded. 
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